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Topography 
Simulation 


using depth scaling 
over an approximate area of 
1350 mm X 850 mm 


oJ 





1 Information 


ave are ae 

A standard 2d topographical map 

aphic maps are based on topographical surveys 
nmed at large scales, These surveys are called 
hical in the old sense, showing a variety of 
vations and landforms. This is in contrast to older 
lastral surveys, which primarily show property and 
nmental boundaries. The first multi-sheet 
aphic map series of an entire country, was 
leted in 1789. The Great ‘Trigonometric Survey of 
started by the East India Company in 1802, was 
for theit successful effort on a larger scale and 
urately determining heights of Himalayan peaks 
n distant viewpoints. Topographic surveys were 
ed first by the military to assist in planning for 
and for defensive emplacements as elevation 
ition was of vital importance in such cases. 
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and 
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However, as they evolved, topographic map_ series 
became a national resource in modern nations in 
planning infrastructure and resource exploitation. 


Topographic maps mostly being represented in two 
dimensional forms lack realistic representation of 
terrain or land relief. Three dimensional presentation of 
the topography on the other hand, helps in detailed 
understanding of land forms and geographic features. 
With the aid of remote sensing satellites, such mapping 
is achieved in reality, However, within the scope of non 
formal education in a Science centre, such attempts can 
help students to acquire deeper knowledge and better 
comprehension. 


A realistic ecosystem projected on top of a material 
surface emphasizes the variation in depth to recreate 
real-time model of a topological map which can reform 
according to the relief of the surface under 
consideration. However, the best part of the setup lies in 
its dynamic nature as it reforms the map as per changing, 
topography. Hence it allows simulating any topography 
of need and thus analyzing the same as required. 





Microsoft Kinect for windows is utilized here as a sensor 
to detect and scale a depth pattern in three dimension 
and thus to convey the data to a set of processing 
algorithm which determines the colour of a layer 
relieved at certain height, to project it back on the same 
surface with the help of a standard LCD projector. 
Starting from Physical modelling of topography to its 
analysis featuring an immediate reformation of the 
modified point cloud helps in understanding realistic 
behavior of a system targeted for simulation. In due 
course of the analysis we will plunge deeper into the 
working of the system and see how the challenges 
involved were overcome. 
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The topography simulator can be described as a 
geographic map featuring changes in height, which can 
be manipulated and presented at a smaller scale yet with 
considerable degree of accuracy. As the map portrays 


detailed and accu 





ute graphic presentation of natural 


und geographical features of the land it can even be 





generalized as a smaller scale chronographic map 
simulator where larger part of the ground is covered 


with reference to the elevation of the region 


« Transformation from the 
regime of two dimensional 
maps to three dimensional 
mapped models 





Two dimensional topographic maps in most of the 
cases fail to realistically portray the depth perception. A 
three dimensional presentation of the same help 
students or professionals understand the land forms 











21) topography 





assuring a higher level of comprehension. ‘The maps, 
mostly being static in nature does not allow instant 
simulation of changes and thus to project implications 
so introduced. However the simulator can make a shift 
from the regime of static 2D maps to the era of dynamic 
3D models while maintaining a great level of accuracy 


». Utilization of Kinect in the 
field of education more 

specific to the science centre 
environment of self learning 








Kinect, initially introduced to the consumers, was 


intended for gaming and solely entertainment 


However, its remarkable combination of hardware, 
firmware and SDK allowed us to de ploy the same for 
innovative projects. Keeping in view the dual nature of 
the system, within the arena of informal education in 
science centre and museums, it has evolved to be a 
gadget for futuristic presentation. Hence using Kinect 
under the realm of education is not as common as using 
the same in the field of gaming & entertainment, 
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pth image processing in 
ontext of surface profile 


vice was initially used for skeletal tracking and 
ig utilized for the purpose of determining 
tures, the impact of using depth frames solely 
been explored to this extent. However the 
y of the device to remotely sense undulation of 
es and capacity of .net framework to implement 

¢ within the scope of windows presentation 
lation is unique to some extent. Although the 
‘olely depends on infrared spectrum which 
s gets affected by random interferences even 
it applications, a carefully selected band and 
filtration of data may help to prepare and 
areal life environment with much ease. 


reat features of this system is its dynamic 
and hence it allows simulation of unstable 
§ prone to land slide in hilly areas and soil 
f near sea shores. Hence it may help even 
onal planners and architects to simulate a 
aS per practical situation and understand the 
es before proceeding with final construction at 







y blocks of the system can be divided under 
sub-heads which work hand in hand despite 
as per their region of application. 





IREmitter Color Sensor 





Microphone Array 


1. Microsoft Kinect for Windows 


The innovative technology behind Kinect is a 
combination of hardware and software contained 
within the sensor itself that can be added to any existing 
computer, The Kinect sensor is a flat black box that sits 
ona small platform, usually horizontally; however in the 
application under consideration, it is mounted vertically, 
There's a trio of hardware innovations working together 
within the Kinect sensor: 


| Color VGA video camera - This video camera aids in 
facial recognition and other detection features by 
detecting three color components: red, green and blue. 
Microsoft calls this an "RGB camera" referring to the 
color components it detects. 








| Depth sensor - An infrared projector and a 
monochrome CMOS (complimentary metal-oxide 
semiconductor) sensor work together to "see" the room 
in 3-D regardless of the lighting conditions. 


| Multi-array microphone - This is an array of four 
microphones that can isolate the voices of the players 
from the noise in the room. This allows the player to bea 
few feet away from the microphone and still use voice 
controls, 


However within the scope of our discussion, we'll 
concentrate only on the depth sensor which generates 
depth data at a rate of 30 frames per second with a 
maximum resolution of 640 by 480 pixels. Unlike a web- 
camera, the depth sensor generates depth frames 
comprised of pixels which indicate distance between the 
sensor and the obstruction at a precision of 
approximately 3 millimeter within the range of 800 
millimeter and 2000 millimeter, Still the hardware, 
though, would be erroneous without the breakthrough 
software that makes use of the data it gathers. The depth 








IR Depth Sensor 


J Tilt Motor 
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Distance from sensor (m) 
o4 8 ee 8 
Default 
Mode 
Near 
Mode 
HB Unknown [Teo Near Mi T00 Far HE Norma Volues 


data being received by the sensor is classified 
according to the following system — 


me so received used to 
in it where 13 bit: 
the depth or distance of ob 


Bach pixel of the depth fra 
have 16 bi 








e used to represent 
struction from sensor and 
left three for holding active player index. The details 
of the same may be presented as below— 








Once depth data is received from the sensor over the 


Property 


0000 1010 1401 0111 


l J vie] 


Depth Bits Player Index 





high speed USB to the computer, depth bits from 
each pixel is filtered and further analyzed to map the 
tes Next the 
challenge lies in minimizing the error and non 
lineari 








ain and undulations on the surfac 





so received. In the 





y present in the pixe 






process of discussing the details of the application 
software we will understand how the same is 
minimized, Overall specification of depth data can 


be presented as below — 


Value 





Angular Field of View 


57° horz., 43° vert. 





Framerate 


approx. 30 Hz 





Nominal spatial range 


640 x 480 (VGA) 











Nominal spatial resolution (at 2m distance) 3mm 
Nominal depth range 0.8 m - 3.5 m 
Nominal depth resolution at 2n1 distance lcm 





Deuce connection type 


2. An application software to analyze depth data 
received from the sensor— 


Once depth data is received in the form of depth 
frames which are 640 by 480 pixels generated at a rate 
of 30 frames per second by the Kinect, the first job is to 
filter the data, Broadly three types of filtration is 
adopted for realistic sensing, We will hence discuss the 
methods involved in such filtration routines one after 
the other, 


Error and imperfection in the Kinect data may 
originate from three main sources: 


- The sensor; 





- Measurement setup; 





- Properties of the object surface. 


USB (+ external power) 
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The sensor errors, for a properly functioning device, 
Mainly refer to inadequate calibration and inaccurate 
Measurement of disparities. Inadequate calibration 
And/or error in the estimation of the 
parimeters lead to systematic error in the object 
Gootdinates of individual points. Such systematic errors 
Gin be eliminated by a proper calibration as described in 
the previous section. Inaccurate measurement of 
Ukparities within the correlation algorithm and 
Particularly the quantization of the disparities also 
jnflence the accuracy of individual points. 


300 





calibration 


Birors caused by the measurement setup are mainly 








Plated to the lighting condition and the imaging 
geometry. The lighting condition influences the 


Correlation and measurement of disparities. In strong 
light the laser speckles appear in low contrast in the 
infrared image, which can lead to outliers or gap in the 
Fesulting point cloud. The imaging geometry includes 





Raw Depth Image 


the distance to the object and the orientation of the 
object surface relative to the sensor. The operating range 
of the sensor is between 0.5 m to 5.0 m according to the 


specifications, and, as we will sce in the following 





section, the random error of depth measurement 
increases with increasing distance to the sensor. Also, 
depending on the imaging geometry, parts of the scene 
may be occluded or shadowed. In Figure 1, the right side 


of the box is occluded as it cannot be seen by the 





infrared camera though it may have been illuminated by 





the laser pattern. The left side of the box is shadowed 
because it is not illuminated by the laser but is captured 
. Both the 


shadows appear as gaps in the point cloud, 


in the infrared imag occluded areas and 








‘The properties of the object surface also impact the 


measurement of points. As it can be seen in Figure 1 
smooth and shiny surfaces that appear overexposed in 





the infrared image (the lower part of the box) impede 
the measurement of disparities, and result ina gap in the 
point cloud. 


Removal of errors in depth measurement 


The first step in removing depth data discrepancy is 





making a space average of the pixels those cannot be 
identified by the sensor and hence marked as unknown. 
However precaution must be taken to differentiate 
between actual unknown values and unknown values 
adopted by the sensor by mistake. ‘The process of 
segregation is calculated in a way that checks if an 
unknown pixel is surrounded by known pixel values of 
similar nature and if it happens a space averaged value 
of known pixels is placed at the position of the incorrect 
or unknown pixel. Shown below is the snap of a depth 


frame with minimized intermittent unknown pixels. 





Smooth Depth Image 
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Stray mater bodies without filtration 


Instead of implementing complex algorithm to correct 





the depth data so received at the cost of valuable system 





resource, a dynamic adaptation technique is introduced. 
Before we started making complex reliefs on the 


surface, it is made as flat as possible, As kinect can 





measure depth right beneath the sensor with 





considerable accuracy, an approximate rectangle of 100 





by 100 pixels is created at the desired coordinate and all 
the pixel values are averaged at that area. Once the 
average pixel values are received, it records deviation of 
each value from the expected values in each pixel and 
record the data in a binary file. Serialization technique is 
introduced in the code to avoid bottleneck in data flow. 
Once the error data is created and saved at an external 
file, in order to remove anomalies in depth measurement 
that error file is loaded and introduced to the execution 
flow so that each time a new depth frame arrives, those 
pre recorded error values are subtracted or added with 





real time depth values. Thus a considerable accuracy is 


achieved in measurement 


Another vital step in minimizing depth errors is 


introduction of time averaging, In order to remove 
spurious responses, an average value of pixels over the 
last 10 frames is calculated and stored in a buffer. At the 
time of processing a certain depth pixel, it is compared 
with time average of depth over the last 10 frames. If the 
ins 





deviation is within a tolerable limit, the data tema 
unaltered however a steep change in the depth received 
triggers the filter to bring the pixel value back into its 
average depth calculated over last few frames, Once the 





Removal of stray water bodies after filtration 





frameis filtered to a desired level, according to the depth 


of each pixel, an image is formed. Each pixel of the 





image so formed used to indicate distance between the 
sensor and the obstruction and hence is colored as per 
standard topographical convention. However due to 
limitation of the sensor itself any value below 400 
millimeter is ignored and gets marked as unknown. Bach 
time frame arrives from the sensor, the code behind 
checks if the frame is empty and if so happens it waity 


for the next frame to 4 rand discard the current 





ppes 
frame, Otherwise, the frame is taken for further 
processing, Two vital methods involved in the process 


are creating a boundary in between the demarcation of 


different layers and mixing up two different colors 
with slowly changing gradient. Once the pixel array of 
the color image is finalized, the entire image is 
transferred to a bitmap source and the source property 
of the ultimate image to be projected is linked with the 


bitmap source so formed 


Once the final image is ready for projection and appears 


on the main window, all that is left out is remapping and 





rescali 





g the image. Rescaling assures that the image 





would not cross the boundaries of projection and 
intended interaction area. Whereas rescaling ascertain 
that peaks and crests are matching with the two 


dimensional color image so that it wraps precisely 





undulations of the surface. Failing to accomplish both 
above steps would result in wrong indication of depth 


and heights with unintended colors. 


ce Communication 
















Code for the Simulation 


j = 640; j <= (depthFrame.PixelDataLength - 640); j++) 


int linenum = j / 640; 
int elementNum = j % 640; 


if (((LineNum % 2) == @) && ((elementNum % 2) == @)) 
if ((depthData@[j] >> 3) == myKinect.DepthStream.UnknownDepth) 


if(((depthData@[j - 1] >> 3) != myKinect.DepthStream.UnknownDepth) && 
((depthData@[j + 640] >> 3) != myKinect.DepthStream.UnknownDepth) && 
((depthData@[j - 640] >> 3) != myKinect.DepthStream.UnknownDepth) ) 
{ 


int depthValue = ((depthData@[j - 1] >> 3) + (depthData@[j + 640] >> 3) + 
(depthData@[j - 640] >> 3)) / 3; 
depthValue = depthValue << 3; 
(depthData@[j]) = (short)depthValue; 











ng (DepthImageFrame depthFrame = e.OpenDepthImageFrame()) 
if (depthFrame == null) return; 


frameCounter++; 
if (frameCounter >= 900) frameCounter = 0; 


‘if (depthData@ == null) depthData® = new short[depthFrame.PixelDataLength]; 
if (depthDatal == null) depthDatal = new short[depthFrame.PixelDataLength]; 





‘if (depthData2 null) depthData2 = new short[depthFrame.PixelDataLength]; 
if (depthData3 == null) depthData3 = new short[depthFrame.PixelDataLength]; 
‘if (depthData4 null) depthData4 = new short[depthFrame.PixelDataLength]; 
if (depthDatas null) depthData5 = new short[depthFrame.PixelDataLength]; 
if (depthDatas null) depthData6 = new short[depthFrame.PixelDataLength]; 
‘if (depthData7 null) depthData7 = new short[depthFrame,PixelDataLength]; 


ou unmet 








if (depthColorImage == null) depthColorImage = new byte[depthFrame.PixelDataLength * 4]; 
‘if (colorImage == null) colorImage = new int[depthFrame.PixelDataLength]; 


if ((frameCounter % 8) == @) depthFrame.CopyPixelDataTo(depthDatag) ; 
if ((frameCounter % 8) == 1) depthFrame.CopyPixelDataTo(depthDatal) ; 
if ((frameCounter % 8) 2) depthFrame.CopyPixelDataTo(depthData2) ; 
if ((frameCounter % 8) == 3) depthFrame.CopyPixelDataTo(depthData3) ; 
if ((frameCounter % 8) 4) depthFrame.CopyPixelDataTo(depthData4) ; 
if ((frameCounter % 8) 5) depthFrame.CopyPixelDataTo(depthDataS) ; 
if ((frameCounter % 8) == 6) depthFrame.CopyPixelDataTo(depthDataé); 
if ((frameCounter % 8) == 7) depthFrame.CopyPixelDataTo(depthData7) ; 
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for (int i = @; i < depthFrame.PixelDataLength; i++) 
{ 


int depthValue = 0; 
if (depthDatae != null && depthDatal != null && depthData2 != null 
&& depthData3 != null && depthData4 != null) 
{ 
depthValue = (int)((depthdata@[i] + depthDatal[i] + depthdata2[i] 
+ depthdata3[i] + depthData4[i] + depthDataS[i] + depthData6[i] + depthbata7[i]) / 8); 


} 
depthDatae[i] = (short)depthValue; 












if (error == null) error = new short[depthFrame.PixelDataLength]; 
for (int h = @; h < depthFrame.PixelDataLength; h++) 
{ int depth = depthData@[h] >> 3; 
error[h] = (short)(140@ - depth); 
if (recordFlag == true) 
: storeCalibration.Record(error); 


storeCalibration.Stop(); 
recordFlag = false; 





class errorRecorder 


{ 
BinaryWriter writer; 
Stream recordStream; 


public void Start(Stream stream) 
{ 
recordStream = stream; 
writer = new Binarywriter(recordStream) ; 


} 
public void Record(short[] error) 


if (writer == null) 
throw new Exception("You must call Start before calling Record"); 


BinaryFormatter formatter = new BinaryFormatter(); 
formatter. Serialize(writer.BaseStream, error); 
} 


public void Stop() 


if (writer == null) 
throw new Exception("You must call Start before calling Stop"); 


writer.Close(); 
writer.Dispose(); 


recordStream.Dispose(); 
recordStream = null; 





“errorRetriever 













public short[] retrieveError(Stream stream, int len) 
short[] retrievedError = null; 


BinaryReader reader = new BinaryReader(stream); 
BinaryFormatter formatter = new BinaryFormatter(); 


retrievedError = new short[len]; 
retrievedError = (short[])formatter.Deserialize(reader.BaseStream) 5 


reader.Dispose(); 
return (retrievedError); 


code shows the sample code for top most two levels) 


fe == myKinect.DepthStream.UnknownDepth) | | 
== mykinect.DepthStream.TooNearDepth) || (depthvalue == mykinect.DepthStream.TooFarDepth) ) 


alpha = 255; 
cd 


fs" Level 13 ***/ 
‘if (depthvalue >= 1190 && depthValue < 1205) 


seedR = 1205 - depthValue; 
seedR “= 75 
seedR = (135 + seedR) & 2555 


seedG = 1205 - depthValue; 
seedG “= 75 
seedG = (135 + seedG) & 255; 


seedB = 1205 - depthValue; 
seedB *= 7; 
seedB = (135 + seedB) & 255; 


blue = (byte)seedB; 
green = (byte)seedG; 
red = (byte)seedR; 
alpha = 255; 
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3. Standard LCD projector mapped to match reliefs 
on the surface — 


A standard projector equipped with 3 LCD technologies 
is utilized to project the image so formed on top of the 
material surface. The projector with an intensity of 4200 
ANSI Lumen placed at a height of approximately 1750 
millimeters is deployed typically for this application. 








Placement of the projector used to be at an edge of the 
table so as to confirm the projection doesn't get 
hindered with the sensor placed right on top of the area 
of interest. 


Al 





n involved 





1. Wait for main application window to get loaded 


2. Initialize kinect and enable depth frame with desired 
resolution 


3. Initialize depth frame ready event and wait for a new 
depth frame to arrive 


4. Lf depth frame is null, return else process depth frame 





5. Copy data from data frame to short pixel array 
6. Initiate filtration of depth data— 

a, Perform pixel filtering 

b. Remove distortions in depth measurement 

c. Perform time averaging 


cate new instance of color image pixel array of 
length 





8. Pick up each pixel one after another by looping 
9, If the pixel is undefined, don't process it further 


10. If the pixel fall within the range of definition, relate 
the corresponding pixel in colorimage 


11. Set the value of Red, Green, Blue and Alpha bytes 
individually as per definition 


12. Once each pixel of the depth frame is processed, 
form the color image frame 











13. Loop through all the pixels in the color image 
formed and detect boundaries 


14. Create line of demarcation between sea levels 


15, Apply gradient brush in between levels if the sa 
above sea level. 





16, Once final image is ready for projection processit 
rescaling and remapping, 


17. Rescale the width and height of the color imaget 
match the area of interest 


18. Black out portions beyond the boundary 
projection to hide unwanted interaction 


19. Remap the color image by adjusting the X and] 
coordinate of the image 


20, Match undulations of the surface precisely with 
projected image 


21. Once rescaled and remapped, create a new instan 
of awritable bitmap 


22. Set the source of the finally projected image to 
writable bitmap 





opy pixels from color image to the writable bit 
to getit linked to the final image 


24. Compile and run the code to see performance 
25. Wait for 10 seconds for stabilization of depth data 


26. Make the surface under observation as flat 
possible 


27. Initiate calibration and save calibration data 


28. Create reliefs and observe if the changes are getti 
reflected 


29. Load calibration data and see the difference 
tracking 


30. Once finished with all above steps, proceed for fi 
setup 
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STANDARD FILTER NOISE 
DESKTOP PC 
RUNNING ON 
64-BIT 
WINDOWS 7 
EQUIPPED 
WITH C#.NET 
PACKAGE 


1, Remove Nonlinearity in response of 
the filter itself 
2. Time averaging of pixels to remove 








spurious response 
3. Guessing values of unidentified pixels 
surrounded by known pixels 

























DEPTH FRAM! 
640 X 480 PIX! 
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COPYING PIXELS FORMATION OF PIXEL ARRAYS BY 

OF THE ARRAY TO |< TINTING INDIVIDUAL PIXELS AS 

A BITMAP SOURCE PER UNDULATION OF SURFAC 
USB 2.0 OR ABOVE U 


COMPARISON WITH 
RENCE PATTERN AND 


REMAPPING THE IMAGE AS PER 
WIDTH AND HEIGHT OF 
INTERACTION SPACE 
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rm ISCALING THE IMAGE TO MATCH 
anus 34 AND PEAKS OVER THE 
PROJECTOR SURFACE PRECISELY 



























SO FORMED 








RGB CAME 
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SURFACE UNDER CONSIDERATION 

















Schematic Drawing of the Setup — 
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Initially it may look like a fun tech demonstration; 


however intended use may cover serious applications in 
the field of simulation. Starting with rain water flow 
path simulation to landscaping and determining 
unstable land forms prone to erosion and landslides may 
well be understood with such system. It may too play a 
vital role in making people understand formation of 
Above all, its 


different geogra 





phical landscape 
application in the field of non formal education in 
science centre and museums would be boundless. 


The authors owe their gratitude to the Director NSCM, 
who compelled us to attempt this project. Not with 
standing the initial problems, that we faced in achieving 
the desired result, the final out come was truly 





remarkable, which we witnessed at the Indian Science 


Congress 2015, where this exhibit was first displayed. 
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